Sensor Porting Guide

1. SENSOR PORTING RELATED HARDWARE INFORMATION¶

This document describes how to porting SigmaStar sensor driver including basic hardware knowledge, software control sensor API (sensor handle), sensor interface API (Sensor-IF) of SigmaStar and software porting flow. In this document, we take the sensor driver of imx415/imx335/sc2315/sc2335 as examples. You can follow the instructions in this document to complete the sensor porting step by step.

First we introduce the relevant knowledge of hardware and corresponding source code to make it easier to understand for you.

1.1. Sensor power sequence¶

Check the power-on sequence spec in the sensor datasheet. Please follow below power sequence 1.1V > 1.8V > 2.9V as the imx415 sensor datasheet required.

Figure 1: the IMX415 sensor power sequence spec

Table 1: the IMX415 sensor power sequence timing table

• Corresponding source code

  You can control sensor power-on and reset pins through “handle->pCus_sensor_poweron” in the sensor driver handle initial function. If you need to modify sensor power sequence timing, please check the characteristics of sensor and hardware design.

  static int pCus_poweron(ms_cus_sensor *handle, u32 idx)  {
      ISensorIfAPI *sensor_if = &handle->sensor_if_api;
      //Sensor power on sequence
      sensor_if->PowerOff(idx, handle->pwdn_POLARITY);   // Powerdn Pull Low
      sensor_if->Reset(idx, handle->reset_POLARITY);     // Rst Pull Low
      sensor_if->SetIOPad(idx, handle->sif_bus, handle->interface_attr.attr_mipi.mipi_lane_num);
      sensor_if->SetCSI_Clk(idx, CUS_CSI_CLK_216M);
      sensor_if->SetCSI_Lane(idx, handle->interface_attr.attr_mipi.mipi_lane_num, ENABLE);
      sensor_if->SetCSI_LongPacketType(idx, 0, 0x1C00, 0);
      if (handle->interface_attr.attr_mipi.mipi_hdr_mode == CUS_HDR_MODE_SONY_DOL)
          sensor_if->SetCSI_hdr_mode(idx, handle->interface_attr.attr_mipi.mipi_hdr_mode, 1);
  
      sensor_if->PowerOff(idx, !handle->pwdn_POLARITY);
      //Sensor board PWDN Enable, 1.8V & 2.9V need 30ms then Pull High
      SENSOR_MSLEEP(31);
      sensor_if->Reset(idx, !handle->reset_POLARITY);
      SENSOR_UDELAY(1);
      sensor_if->MCLK(idx, 1, handle->mclk);
      SENSOR_DMSG("Sensor Power On finished\n");
      return SUCCESS;
  }
  

NOTE: Below is SigmaStar Sensor board schematic of power circuit. 1.1V, 1.8V, 2.9V power will pull high after power-on enable. First 1.1V, then 1.8V and 2.9V, you can use an oscilloscope to measure it on your sensor board.

Figure 2: the IMX415 sensor power circuit

1.2. Sensor IC slave address check¶

SigmaStar SoC needs to communicate with a sensor through I2C, please check the sensor I2C slave address first.

Table 2: the IMX415 sensor slave id table

NOTE: Check the sensor board schematic of I2C slave mode Option

Figure 3: the IMX415 sensor board I2C slave mode

• Corresponding source code

  The sensor driver I2C Setting in Marco define

  ////////////////////////////////////
  // I2C Info                         //
  ////////////////////////////////////
  #define SENSOR_I2C_ADDR     0x34        //I2C slave address
  #define SENSOR_I2C_SPEED    300000      //I2C speed, 60000~320000
  //#define SENSOR_I2C_CHANNEL   1        //I2C Channel
  //#define SENSOR_I2C_PAD_MODE  2        //Pad/Mode Number
  #define SENSOR_I2C_LEGACY  I2C_NORMAL_MODE  //usually set CUS_I2C_NORMAL_MODE,  if use old OVT I2C protocol=> set CUS_I2C_LEGACY_MODE
  #define SENSOR_I2C_FMT     I2C_FMT_A16D8 //CUS_I2C_FMT_A8D8, CUS_I2C_FMT_A8D16, CUS_I2C_FMT_A16D8, CUS_I2C_FMT_A16D16
  

NOTE: Sensor-IF of I2C control Interface in handle initial function

handle->i2c_cfg.mode     = SENSOR_I2C_LEGACY;    //(CUS_ISP_I2C_MODE) FALSE;
handle->i2c_cfg.fmt      = SENSOR_I2C_FMT;       //CUS_I2C_FMT_A16D16;
handle->i2c_cfg.address  = SENSOR_I2C_ADDR;      //0x34;
handle->i2c_cfg.speed    = SENSOR_I2C_SPEED;     //300000;

1.3. Sensor MCLK support¶

Each sensor can support in different MCLK. That the MCLK list table that we can provide is as follows. You can find the MCLK corresponding to the sensor from the initialization settings provided by the sensor vendor. The sensor MCLK Setting in Marco define

• Corresponding source code

  ////////////////////////////////////
  // MCLK Info                        //
  ////////////////////////////////////
  /*******I5/MACARON Support MCLK List*******
  *    CUS_CMU_CLK_27MHZ   / CUS_CMU_CLK_21P6MHZ  / CUS_CMU_CLK_12MHZ,
  *    CUS_CMU_CLK_5P4MHZ  / CUS_CMU_CLK_36MHZ    / CUS_CMU_CLK_54MHZ,
  *    CUS_CMU_CLK_43P2MHZ / CUS_CMU_CLK_61P7MHZ  / CUS_CMU_CLK_72MHZ,
  *    CUS_CMU_CLK_48MHZ    / CUS_CMU_CLK_24MHZ    / CUS_CMU_CLK_37P125MHZ,
  ******End of Support MCLK List*******/
  #define Preview_MCLK_SPEED           CUS_CMU_CLK_27MHZ
  #define Preview_MCLK_SPEED_HDR_DOL  CUS_CMU_CLK_37P125MHZ
  

  NOTE: The sensor MCLK control Interface in handle initial function

  ////////////////////////////////////
  //    mclk                        //
  ////////////////////////////////////
  handle->mclk                   = UseParaMclk(SENSOR_DRV_PARAM_MCLK());
  

2. Sensor porting relevant software control¶

After introducing the relevant knowledge of hardware and corresponding source code, we will introduce software control API (sensor handle) and sensor interface API (Sensor-IF) of SigmaStar in this part.

2.1. Sensor driver support mode define¶

Register the sensor support behavior mode. For example, the IMX415 sensor supports linear and HDR mode, you can refer to the IMX415 sensor driver and fill it in define of “SENSOR_DRV_ENTRY_IMPL_END_EX”.

SENSOR_DRV_ENTRY_IMPL_END_EX(  IMX415_HDR,
                            cus_camsensor_init_handle,
                            cus_camsensor_init_handle_hdr_dol_sef,
                            cus_camsensor_init_handle_hdr_dol_lef,
                            imx415_params
                            );

Table 3: the sensor driver “SENSOR_DRV_ENTRY_IMPL_END_EX” define

2.1.1. Introduce “SENSOR_DRV_IMPL_END_EX”¶

You can check macro define “SENSOR_DRV_ENTRY_IMPL_END_EX” more detail in the path “driver/SensorDriver/drv/inc/drv_sensor_common.h”. LinearEntry has a group of private data set, HdrSefEntry and HdrLefEntry have another a group of private data set.

#define SENSOR_DRV_ENTRY_IMPL_END_EX (Name,LinearEntry,HdrSefEntry,HdrLefEntry,PrivateDataType)\
static PrivateDataType* g_pData[2][MAX_CAMID_LEN] = { {0,0,0,0}, {0,0,0,0} };\
int Name##_init_driver(unsigned char chmap)\
{\
    int nCamID=0;\
    /*To avoid compile warning*/\
    void* p0 = LinearEntry;\
    void* p1 = HdrSefEntry;\
    void* p2 = HdrLefEntry;\
    if(DrvSensorHandleVer(CUS_CAMSENSOR_HANDLE_MAJ_VER, CUS_CAMSENSOR_HANDLE_MIN_VER)==FAIL)\
    return FAIL;\
    if(DrvSensorIFVer(CUS_CAMSENSORIF_MAJ_VER, CUS_CAMSENSORIF_MIN_VER)==FAIL)\
    return FAIL;\
    if(DrvSensorI2CVer(CUS_CAMSENSOR_I2C_MAJ_VER, CUS_CAMSENSOR_I2C_MIN_VER)==FAIL)\
    return FAIL;\
    for(nCamID=0;nCamID<MAX_CAMID_LEN;++nCamID)\
    {\
        if((chmap>>nCamID)&0x1)\
        {\
            if(p0){\
                void* pData = CamOsMemAlloc(sizeof(PrivateDataType));\
                memset(pData,0,sizeof(PrivateDataType));\
                DrvRegisterSensorDriverEx(nCamID, LinearEntry,pData);\
                DrvRegisterSensorI2CSlaveID(nCamID, (i2c_slave_id>>(nCamID*8))&0xFF);\
                SENSOR_DMSG("Connect %s linear to sensor pad %d\n",__FUNCTION__, nCamID);\
                g_pData[0][nCamID] = pData;\
            }\
            if(p1||p2){\
                void* pData = CamOsMemAlloc(sizeof(PrivateDataType));\
                memset(pData,0,sizeof(PrivateDataType));\
                if(p1){\
                    DrvRegisterPlaneDriverEx(nCamID, 1, HdrSefEntry,pData);\
                    DrvRegisterPlaneI2CSlaveID(nCamID, 1, (i2c_slave_id>>(nCamID*8))&0xFF);\
                    SENSOR_DMSG("Connect %s SEF to vif sensor pad %d\n",__FUNCTION__, nCamID);\
                }\
                if(p2){\
                    DrvRegisterPlaneDriverEx(nCamID, 0, HdrLefEntry,pData);\
                    DrvRegisterPlaneI2CSlaveID(nCamID, 1, (i2c_slave_id>>(nCamID*8))&0xFF);\
                    SENSOR_DMSG("Connect %s LEF to sensor pad %d\n",__FUNCTION__, nCamID);\
                }\
                g_pData[1][nCamID] = pData;\
            }\
        }\
    }\
    return 0;\
}\

2.2. Sensor driver handle function¶

Next we will introduce and explain handle API in the linear and HDR handle initial functions. We can control, set or read any state of the sensor through the handle API. Here we take the sensor driver of imx415 as an example. You can step through the sensor porting of linear and HDR mode according to the instructions in this document.

2.2.1. Handle of IC API¶

Set I2C hardware information

Table 4: handle of I2C API define

2.2.2. Handle of MCLK API¶

Set the sensor MCLK information

Table 5: handle of MCLK API define

2.2.3. Handle of Sensor model id API¶

Set sensor name and mode information

Table 6: handle of model id API define

2.2.4. Handle of sensor interface and output hardware information¶

Table 7: handle of Sensor interface and output hardware information API define

Parameter	Describe	NOTE
handle->pCus_sensor_poweron	Fill Up Sensor IF Info!!! Register set sensor power-on sequence function.	Function ref : pCus_poweron
handle->pCus_sensor_init	Register load sensor initial setting function.
handle->pCus_sensor_post_init	Set and enable SoC mipi csi status.	Special case is like SmartSens product SC4236/SC2315
handle->isp_type	Register ISP type (not used in MACARON)	SENSOR_ISP_TYPE ISP_SOC, /**\< Not support */ ISP_EXT, /**\< sensor without built-in ISP */
handle->sif_bus	Register set sensor interface bus type	SENSOR_IFBUS_TYPE CUS_SENIF_BUS_PARL = 0, /**\< sensor data bus is parallel */ CUS_SENIF_BUS_MIPI = 1, /**\< sensor data bus is mipi */ CUS_SENIF_BUS_BT601 = 2, CUS_SENIF_BUS_BT656 = 3, CUS_SENIF_BUS_BT1120 = 4,
handle->data_prec	Register set sensor output raw data precision	SENSOR_DATAPREC CUS_DATAPRECISION_8 = 0, /**\< raw data precision 8bits */ CUS_DATAPRECISION_10 = 1, /**\< raw data precision 10bits */ CUS_DATAPRECISION_16 = 2, /**\< raw data precision 16bits */ CUS_DATAPRECISION_12 = 3, /**\< raw data precision 12bits */ CUS_DATAPRECISION_14 = 4, /**\< raw data precision 14bits */
handle->data_mode	Register set sensor bayer raw (8/10-bits) to 12-bits mode (Not used in MACARON)	SENSOR_DATAMODE CUS_SEN_8TO12_7074, CUS_SEN_8TO12_7000, CUS_SEN_8TO12_114118, CUS_SEN_8TO12_11400, CUS_SEN_10TO12_9098, CUS_SEN_10TO12_9000, CUS_SEN_10TO12_1121110, CUS_SEN_10TO12_11200
handle->bayer_id	Register set sensor bayer raw start id	SENSOR_BAYERID CUS_BAYER_RG = 0, CUS_BAYER_GR, CUS_BAYER_BG, CUS_BAYER_GB,
handle->RGBIR_id	Register set sensor RGB-IR start id #If NOT RGB-IR sensor, please set CUS_RGBIR_NONE	SENSOR_RGBIRID CUS_RGBIR_NONE = 0, CUS_RGBIR_R0 = 1, CUS_RGBIR_G0 = 2, CUS_RGBIR_B0 = 3, CUS_RGBIR_G1 = 4, CUS_RGBIR_G2 = 5, CUS_RGBIR_I0 = 6, CUS_RGBIR_G3 = 7, CUS_RGBIR_I1 = 8
handle->orient	Register get sensor image mirror-flip default status	SENSOR_ORIT CUS_ORIT_M0F0, /**\< mirror, flip unchanged */ CUS_ORIT_M1F0, /**\< mirror changed */ CUS_ORIT_M0F1, /**\< flip changed */ CUS_ORIT_M1F1, /**\< mirror and flip changed */
handle->snr_pad_group	Register sensor pad group	CUS_SENSOR_PAD_GROUP_A = 0, CUS_SENSOR_PAD_GROUP_B = 1, CUS_SENSOR_PAD_GROUP_A = 0, CUS_SENSOR_PAD_GROUP_B = 1,
handle->snr_pad_mode	Register sensor master and slave mode	CUS_SENSOR_MASTER_MODE = 0, CUS_SENSOR_SLAVE_MODE = 1,
handle->ir_only_fmt	Register sensor ir format status
For HDR (MIPI) Interface
handle->interface_attr.attr_mipi.mipi_lane_num	Register set MIPI lane number	Default : 4
handle->interface_attr.attr_mipi.mipi_data_format	Register set MIPI data format	CUS_SEN_INPUT_FORMAT_YUV422, CUS_SEN_INPUT_FORMAT_RGB,
handle->interface_attr.attr_mipi.mipi_yuv_order	Register set MIPI yuv order	For yuv sensor only
handle->interface_attr.attr_mipi.mipi_hsync_mode	Register set HDR virtual channel0 hsync mode	SENSOR_MIPI_HSYNC_MODE PACKET_HEADER_EDGE1 = 0, /**\< packet header edge */ PACKET_HEADER_EDGE2 = 1, /**\< line end edge */ PACKET_HEADER_EDGE3 = 2, /**\< line start edge */ PACKET_FOOTER_EDGE = 3, /**\< packet footer edge */
handle->interface_attr.attr_mipi.mipi_hdr_mode	Register set MIPI HDR mode	CUS_HDR_MODE_NONE = 0, CUS_HDR_MODE_SONY_DOL = 1, CUS_HDR_MODE_DCG = 2, CUS_HDR_MODE_EMBEDDED_RAW8 = 3, CUS_HDR_MODE_EMBEDDED_RAW10 = 4, CUS_HDR_MODE_EMBEDDED_RAW12 = 5, CUS_HDR_MODE_EMBEDDED_RAW16 = 6, //Only for OV2718? CUS_HDR_MODE_LI = 7,
handle->interface_attr.attr_mipi. mipi_sampling_delay	MIPI CLK skip time (ns) (Not used in MACARON)
handle->interface_attr.attr_mipi.mipi_hdr_virtual_channel_num	Register set HDR virtual channel number #Based on each sensor VC mode header define	For example imx415: Long Exposure Frame : 0 Short Exposure Frame : 1

2.2.5. Handle of signal polarity API¶

Set SoC hardware signal polarity

Table 8: handle of signal polarity API define

2.2.6. Handle of sensor resolution capability API¶

Set sensor support and implement output image information.

Table 9: handle of sensor resolution capability API define

2.2.7. Handle of set/get sensor control API¶

We can load sensor initial setting, switch sensor output resolution, sensor mirror or flip and fps set or get through these functions.

Table 10: handle of set/get sensor control API define

2.2.8. Handle of set/get sensor AE control API¶

First we will introduce the sensor driver private data in this part, this purpose is to record the parameters of the sensor without any modification. Here is imx415 as a reference example. The naming can be customized according to each sensor or developer, and there is no mandatory requirement.

Table 11: example for imx415 private data

We can set or get sensor AE function status through this API, including AE shutter time, gain, limit AE setting value and AE update timing, etc.

Table 12: handle of set/get sensor AE control API define

2.2.9. Sensor-IF API¶

SENSOR_IF API for Connect with the sensor driver and the VIF driver, this callback control in below handle control function.

We will introduce and describe this API, more detail you can reference the sensor driver of IMX415.

3. Software porting flow¶

This chapter introduces the process of software porting sensor driver. First, it will explain how to initialize the sensor. Then describe the information required by the sensor driver 3A and control the functions that 3A will call.

3.1. The sensor driver initial flow introduction¶

Figure 4: porting the sensor driver flow

This document introduces the initial process of transplanting the sensor according to the process in the figure above.

3.1.1. Set the sensor information setting¶

These settings are the hardware information of the sensor, we have listed the necessary hardware information, you can refer to the IMX415 / IMX335 sensor driver,

////////////////////////////////////
// Sensor-If Info                 //
////////////////////////////////////
//MIPI config begin.
#define SENSOR_MIPI_LANE_NUM        (lane_num)          //IMX335 Linear mode supports MIPI 2/4 Lane
#define SENSOR_MIPI_LANE_NUM_DOL    (4) //(hdr_lane_num)//IMX335 DOL mode supports MIPI 4 Lane
//#define SENSOR_MIPI_HDR_MODE        (0) //0: Non-HDR mode. 1:Sony DOL mode

#define SENSOR_ISP_TYPE             ISP_EXT             //ISP_EXT, ISP_SOC (Non-used)
//#define SENSOR_DATAFMT             CUS_DATAFMT_BAYER    //CUS_DATAFMT_YUV, CUS_DATAFMT_BAYER
#define SENSOR_IFBUS_TYPE           CUS_SENIF_BUS_MIPI  //CFG //CUS_SENIF_BUS_PARL, CUS_SENIF_BUS_MIPI
#define SENSOR_MIPI_HSYNC_MODE      PACKET_HEADER_EDGE1
#define SENSOR_MIPI_HSYNC_MODE_HDR_DOL PACKET_FOOTER_EDGE
#define SENSOR_DATAPREC             CUS_DATAPRECISION_10
#define SENSOR_DATAPREC_DOL         CUS_DATAPRECISION_10
#define SENSOR_DATAMODE             CUS_SEN_10TO12_9098  //CFG
#define SENSOR_BAYERID              CUS_BAYER_RG         //0h: CUS_BAYER_RG, 1h: CUS_BAYER_GR, 2h: CUS_BAYER_BG, 3h: CUS_BAYER_GB
#define SENSOR_BAYERID_HDR_DOL     CUS_BAYER_RG
#define SENSOR_RGBIRID              CUS_RGBIR_NONE
#define SENSOR_ORIT                 CUS_ORIT_M0F0        //CUS_ORIT_M0F0, CUS_ORIT_M1F0, CUS_ORIT_M0F1, CUS_ORIT_M1F1,
//#define SENSOR_YCORDER              CUS_SEN_YCODR_YC     //CUS_SEN_YCODR_YC, CUS_SEN_YCODR_CY
//#define long_packet_type_enable     0x00 //UD1~UD8 (user define)

////////////////////////////////////
// MCLK Info                        //
////////////////////////////////////
#define Preview_MCLK_SPEED          CUS_CMU_CLK_27MHZ//CUS_CMU_CLK_37P125MHZ//CUS_CMU_CLK_27MHZ
#define Preview_MCLK_SPEED_HDR_DOL  CUS_CMU_CLK_27MHZ

3.1.2. Set the sensor IC setting¶

SoC communicates with sensors mainly through I2C. Therefore, you need to set I2C before sending sensor setting value to the sensor.

Please refer to this document chapter 1.2.

3.2. Set support image resolution information¶

You need to fill in the sensor image information provided by the sensor provider FAE. This information includes preview size, image crop size and site etc. We define this information as a global variable.

////////////////////////////////////
// Image Info                     //
////////////////////////////////////
static struct {     // LINEAR
    // Modify it based on number of support resolution
    enum {LINEAR_RES_1 = 0, LINEAR_RES_2, LINEAR_RES_3, LINEAR_RES_4, LINEAR_RES_5, LINEAR_RES_6, LINEAR_RES_7, LINEAR_RES_8, LINEAR_RES_END}mode;
    // Sensor Output Image info
    struct _senout{
        s32 width, height, min_fps, max_fps;
    }senout;
    // VIF Get Image Info
    struct _sensif{
        s32 crop_start_X, crop_start_y, preview_w, preview_h;
    }senif;
    // Show resolution string
    struct _senstr{
        const char* strResDesc;
    }senstr;
}imx415_mipi_linear[] = {
    {LINEAR_RES_1, {3860, 2250, 3, 20}, {0, 0, 3840, 2160}, {"3840x2160@20fps"} },
    {LINEAR_RES_2, {3096, 2190, 3, 30}, {0, 0, 3072, 2048}, {"3072x2048@30fps"} }, // Modify it
    {LINEAR_RES_3, {3096, 1758, 3, 30}, {0, 0, 3072, 1728}, {"3072x1728@30fps"} }, // Modify it
    {LINEAR_RES_4, {2616, 1974, 3, 30}, {0, 0, 2592, 1944}, {"2592x1944@30fps"} }, // Modify it
    {LINEAR_RES_5, {2976, 1686, 3, 30}, {0, 0, 2944, 1656}, {"2944x1656@30fps"} }, // Modify it
    {LINEAR_RES_6, {2592, 1470, 3, 30}, {0, 0, 2560, 1440}, {"2560x1440@30fps"} }, // Modify it
    {LINEAR_RES_7, {1920, 1080, 3, 60}, {0, 0, 1920, 1080}, {"1920x1080@60fps"} }, // Modify it
    {LINEAR_RES_8, {3864, 2192, 3, 30}, {0, 0, 3840, 2160}, {"3840x2160@30fps"} }, // Modify it
    //{LINEAR_RES_9, {2616, 1974, 3, 25}, {0, 0, 2592, 1944}, {"2592x1944@25fps"} }, // Modify it
};

3.2.1. The sensor power on sequence function¶

Please refer to this document chapter 1.1.

3.2.2. The sensor initial function¶

Here is the initial setting of the sensor through the i2c device, first read the product ID or sensor ID of the sensor device to ensure that the correct sensor is used. Please refer to the function registered by “handle->pCus_sensor_init”.

3.2.3. Select the sensor output resolution function¶

MI will get the image resolution information supported by the sensor driver and select one of the sensor's initial settings.

Please refer to the function registered by “handle->pCus_sensor_SetVideoRes”.

3.3. Implement the sensor control function¶

The functions that the sensor driver needs to support include fps adjustment, AE shutter adjustment, AE gain adjustment, and mirror flip function settings, which we will not describe more detail, because this needs to be implemented according to the characteristics of each sensor.

The following is our control the sensor implement function flow. Relatively, we also need to get the sensor information, please refer to the figure below:

Figure 5: control the sensor implement function flow

Figure 6: get the sensor implement function information or status

3.3.1. AE shutter control and get AE shutter information¶

How to control AE shutter, please refer to the function registered by “handle->pCus_sensor_SetAEUSecs”.

Also, ISP algo or MI need to get AE shutter information, please refer to the function registered by “handle->pCus_sensor_GetAEUSecs” and “handle->pCus_sensor_GetShutterInfo”.

3.3.2. AE gain control and get AE gain information¶

How to control AE gain, please refer to the function registered by “handle->pCus_sensor_SetAEGain”.

Also, ISP algo or MI need to get AE gain information, please refer to the function registered by “handle->pCus_sensor_GetAEGain” and “handle->pCus_sensor_GetAEMinMaxGain”.

3.3.3. Sensor frame rate control and get sensor frame rate information¶

How to control the sensor output frame rate, please refer to the function registered by “handle->pCus_sensor_SetFPS”.

Also, MI needs to get the sensor output frame rate information, please refer to the function registered by “handle->pCus_sensor_GetFPS”.

3.3.4. Sensor mirror-flip control and get sensor mirror-flip information¶

How to control the sensor image mirror-flip, please refer to the function registered by “handle->pCus_sensor_SetOrien”.

Also, MI needs to get the sensor image mirror-flip information, please refer to the function registered by “handle->pCus_sensor_GetOrien”.

3.4. IC execution time control¶

The above AE shutter, AE gain, sensor frame rate and mirror control, we hope everyone can send commands to the sensor through I2C at the same time

So we will implement this using the function "handle->pCus_sensor_AEStatusNotify"

Therefore, we can control the time of I2C send commands to the sensor through two parameters "CUS_FRAME_ACTIVE" and "CUS_FRAME_INACTIVE". Most sensors can be set at the frame start, so the parameter "CUS_FRAME_ACTIVE" can be selected.

Figure 7: I2C execution time

NOTE:

If the porting sensor cannot find out what caused the image abnormality, you can call “return success” directly in the function.

3.4.1. Sensor AE shutter/gain delay count¶

How to set “handle->ae_shutter_delay” and “handle->ae_gain_delay”, please refer to this document chapter 4.2.

3.5. Sensor AE shutter/gain delay count¶

How to set “handle->ae_gain_ctrl_num” and “handle->ae_shutter_ctrl_num”, please refer to this document chapter 2.2.8.

4. Supplyment¶

4.1. Install the Sensor Driver and parameter introduce¶

SigmaStar sensor driver provides several installation parameters that can be used when installing the sensor driver. Here are the parameters and instructions as below:

Table 13: sensor driver install parameter

For example:

If the above parameters are not used when installing the sensor driver, the default values ​​in the sensor driver are used.

4.2. Verify and debug video stream¶

The next step is to verify that the sensor driver can be used normally, so, you need to copy app from “/sdk/verify/mi_demo/out/demo/app” to EVB, I take “prog_cus3a” as an example:

/customer # ./prog_cus3a
use default sensor pad 0, vif dev 0
Use HDR ?
0 not use, 1 use DOL
0
You select not use HDR client [548] connected, module:sensor
index 0, Crop(0,0,3840,2160), outputsize(3860,2250), maxfps 20, minfps 3, ResDesc 3840x2160@20fps
index 1, Crop(0,0,3072,2048), outputsize(3096,2190), maxfps 30, minfps 3, ResDesc 3072x2048@30fps
index 2, Crop(0,0,3072,1728), outputsize(3096,1758), maxfps 30, minfps 3, ResDesc 3072x1728@30fps
index 3, Crop(0,0,2592,1944), outputsize(2616,1974), maxfps 30, minfps 3, ResDesc 2592x1944@30fps
index 4, Crop(0,0,2944,1656), outputsize(2976,1686), maxfps 30, minfps 3, ResDesc 2944x1656@30fps
index 5, Crop(0,0,2560,1440), outputsize(2592,1470), maxfps 30, minfps 3, ResDesc 2560x1440@30fps
index 6, Crop(0,0,1920,1080), outputsize(1920,1080), maxfps 60, minfps 3, ResDesc 1920x1080@60fps
index 7, Crop(0,0,3840,2160), outputsize(3864,2192), maxfps 30, minfps 3, ResDesc 3840x2160@30fps
choice which resolution use, cnt 8

4.2.1. Check frame interval¶

At the same time you can open telnet and enter the following cmd to confirm the VIF and ISP status.

Usually 30fps means that a frame period is 33.3ms.

/ # cat /sys/class/mstar/vif0/vif_ints
== VIF CH-0 ==
Interval(ns)                    : 33309000
VREF_RISING                   : 6147
VREF_FALLING                  : 6149
LINE_CNT_0                    : 2
LINE_CNT_1                    : 2

4.2.2. Check the sensor output stream status¶

Confirm the information of the sensor signal received by VIF device

/ # cat /sys/class/mstar/vif0/vif_info
== VIF CH-0 ==
INTF_MODE = 4
CH_EN: 1
SRC: MIPI0
INPUT_FMT: RGB
PIX_FMT: 12 bits
HDR_EN: 0
HDR_CH: VC0
CROP_EN: 1
CROP_START_X: 0 - 1919
CROP_START_Y: 0 - 1079
PIXEL_CNT: 1920
LINE_CNT: 1080

4.2.3. Check the ISP status¶

If the transmission speed of one line or pixel per frame is too fast, even if the VIF can receive the image data normally, there is no guarantee that the ISP can process it immediately, it causes isp fifo full.

/ # cat /sys/class/mstar/isp0/isp_ints
frame_count                     : 1089
u32FrameIntervalNanoSec         : 33332 us
ISP_FIFO_FULL                   : 0
INT_PIPE1_FIFO_FULL             : 0
ISP_BUSY                        : 1089, 06.952374
ISP_IDLE                         : 1090, 06.950575
RDMA0_DONE                     : 0
RDMA1_DONE                     : 0
WDMA0_DONE                    : 0
WDMA1_DONE                    : 0
WDMA6_LINE_HIT                  : 0
WDMA6_DONE                     : 0
ISPIF_VSYNC                       : 1089, 06.952046, 0
AE_DONE                         : 1088, 06.950279, 1934
AE_DONE period                  : 31565 us

4.3. Verify shutter and gain delay¶

If you want to verify that the sensor AE gain/shutter delay, you can use apitool, also follow the steps below to control the sensor AE gain/shutter function step by step.

STEP 1. You need to execute process of mixer first

/customer # ./mixer -q -n 1          (linear mode)
/customer # ./mixer -q -n 1 -N 3_2    (hdr mode)

STEP 2. Open IQ tool in sdk path “project/tools/apitool”

STEP 3. Host Name is EVB’s IP address, check it then connect to EVB

STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode

Step 5. Adjust “DebugLev” menu and set AE Debug Level from 0 to 64, then you can check you console log

Check the AE message shown on the console

Step 6. Adjust AE sensor gain or shutter manually. At the same time, you can observe the AE message shown on the console.

Observer the value of “CurYx10” change from 308 to 1436

After we set the gain of AE through ISP tool, it will take effect in the next frame. The CurYx10 represents the brightness information of a previous frame. The following figure is an example. The AE gain delay needs to set 2.

Figure 8: AE Y value response time

4.4. Verify shutter and gain linearity¶

If you want to verify that the sensor AE gain/shutter function is normal after porting the sensor AE gain/shutter function finish, you can use apitool. You can follow the steps below to control the sensor AE gain/shutter function step by step.

STEP 1. You need to execute process of mixer

/customer # ./mixer -q -n 1        (linear mode)
/customer # ./mixer -q -n 1 -N 3_2    (hdr mode)

STEP 2. Open IQ tool in sdk path “project/tools/apitool”

STEP 3. Host Name is EVB’s IP address, check it then connect to EVB

STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode

Step 5. Adjust AE sensor gain and shutter manually. At the same time, you can observe image changes through VLC or your debug message.

The IQ Tool has defined upper and lower limit for the AE settings, you can adjust it according to your needs. You can reference Exposure limit table.

4.5. Skip frame feature¶

The sensor driver that has been implemented has imx307. You can refer to the sensor driver of imx307. Here I will briefly explain how to use skip frame function.

Step 1. Add a variable "skip _cnt" inside Params structure of sensor driver

u32 max_rhs1;
    u32 lef_shs2;
    u32 skip_cnt;
    bool dirty;
    bool change;
    I2C_ARRAY tVts_reg[3];
    I2C_ARRAY tGain_reg[3];
    I2C_ARRAY tExpo_reg[3]; 
    I2C_ARRAY tShs2_reg[3];
    I2C_ARRAY tRhs1_reg[3];
    I2C_ARRAY tGain_hdr_dol_lef_reg[1];
    I2C_ARRAY tGain_hdr_dol_sef_reg[1];
    bool orien_dirty;
} imx307_params;

Step 2. Set the number of skip frames in the function that needs to skip frames.

For an example of imx307, in HDR mode, it may flicker when changing fps (from 10 fps to 30 fps) in high brightness environment, so skip 1 frame to avoid flicker.

if(params->expo.expo_lines > 2 * params->expo.vts - params->max_rhs1 -3){
    vts = (params->expo.expo_lines + params->max_rhs1 + 3 + 1) / 2;
    }else{
    vts = params->expo.vts;
    }

    params->expo.vts = vts;
    pCus_SetAEUSecsHDR_DOL_SEF1(handle, params->expo.expo_sef_us);
    params->skip_cnt = 1;
    return SUCCESS;

Step 3. The actual time to skip frame in function of “AESStatusNotify”. Please add the following features. SensorIF will calculate the time to set the VIF mask disable based on the number of skip frames and the current fps time period.

            if(params->skip_cnt){
                sensor_if->SetSkipFrame(handle->snr_pad_group, params->expo.fps, params->skip_cnt);
                params->skip_cnt = 0;
            }
            break;
        default :
            break;
    }
    return SUCCESS;

4.6. Sensor post initial feature (Special case for SmartSens)¶

Since the Smartsens sensor first transmit i2c cmd after power-on, the probability of the sensor mipi csi and clock will be pulled down, it leads to SoC mipi csi abnormal.

Figure 9: Smartsense sensor signal

Currently, the solution is to change the sequence of the sensor power-on and the sensor initialization. After the sensor initialize completed, then enable SoC mipi csi to resolve this problem. Please refer to the following process:

Figure 10: power on sequence flow change